DOCUMENT RESUME 



ED 451 383 



CE 081 531 



AUTHOR 

TITLE 



INSTITUTION 
SPONS AGENCY 

REPORT NO 
PUB DATE 
NOTE 

AVAILABLE FROM 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Sticht, Thomas G. 

Using Telephone and Mail Surveys as a Supplement or 
Alternative to Door-to-Door Surveys in the Assessment of 
Adult Literacy. Working Paper Series. 

Education Statistics Services Inst., Washington, DC. 

National Center for Education Statistics (ED) , Washington, 
DC. 

NCES-WP-2000- 06 
2000-03-00 

60p . 

ED Pubs, P.O. Box 1398, Jessup, MD 20794-1398. Tel: 
877-433-7827 (Toll Free). For full text: 
http : / /www.nces . ed . gov/pubs2000/20006 .pdf . 

Information Analyses (070) -- Reports - Evaluative (142) 

MF01/PC03 Plus Postage. 

Adult Basic Education; Adult Development; * Adult Literacy; 
Cognitive Ability; Comparative Analysis; *Educational 
Assessment; Educational Theories; Literature Reviews; *Mail 
Surveys; Models; National Surveys; Oral Language; Research 
Methodology; *Telephone Surveys; *Test Validity; Trend 
Analysis 

International Adult Literacy Survey; National Adult Literacy 
Survey (NCES) ; Survey Research; Workforce Investment Act 
1998 Title II 



ABSTRACT 



This paper discusses the use of telephone surveys as a 
substitute for, or addition to, the use of door-to-door surveys in assessing 
adult literacy ability. Part 1 addresses issues of validity in adult literacy 
assessment, including the concepts of construct or inferential validity and 
action or use validity. Part 2 is a review of related research that has used 
telephone and mail surveys to assess the cognitive abilities of adults. Part 
3 discusses the theoretical model of literacy that the National Adult 
Literacy Survey (NALS) and International Adult Literacy Survey (IALS) have 
used and how that model can be modified to incorporate a developmental and 
componential model of literacy that also includes the oral language aspects 
of literacy given in the Adult Education and Family Literacy Act of 1998. 

Part 4 addresses several issues about the use of telephone and mail surveys, 
including concerns about using these survey methods in such a way as to make 
continuity in trend data from previous surveys of adult literacy possible. 
Appendixes include 69 references and a summary of an empirical study of the 
proposition that at a certain stage adults' comprehension of language can be 
assessed as well by listening as by reading. (Author/YLB) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



ED 451 383 



NATIONAL CENTER FOR EDUCATION STATISTICS 



Working Paper Series 



Using Telephone and Mail Surveys as a 
Supplement or Alternative to Door-to-Door 
Surveys in the Assessment of Adult Literacy 



Working Paper No. 2000-06 March 2000 



Contact: Sheida White 

Assessment Division 
E-mail: sheida_white@ed.gov 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

This document has been reproduced as 
ceived from the person or organization 
originating it. 



7 ^ 



□ Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 






U. S. Department of Education 

Office of Educational Research and Improvement 



0 



VJb 

<5 



NATIONAL CENTER FOR EDUCATION STATISTICS 



Working Paper Series 



The Working Paper Series was initiated to promote the sharing of the 
valuable work experience and knowledge reflected in these preliminary 
reports. These reports are viewed as works in progress, and have not 
undergone a rigorous review for consistency with NCES Statistical 
Standards prior to inclusion in the Working Paper Series. 



U. S. Department of Education 

Office of Educational Research and Improvement 



U.S. Department of Education 

Richard W. Riley 
Secretary 

Office of Educational Research and Improvement 

C. Kent McGuire 
Assistant Secretary 

National Center for Education Statistics 

Gary W. Phillips 
Acting Commissioner 



The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, 
and reporting data related to education in the United States and other nations. It fulfills a congressional 
mandate to collect, collate, analyze, and report full and complete statistics on the condition of education in 
the United States; conduct and publish reports and specialized analyses of the meaning and significance of 
such statistics; assist state and local education agencies in improving their statistical systems; and review 
and report on education activities in foreign countries. 

NCES activities are designed to address high priority education data needs; provide consistent, reliable, 
complete, and accurate indicators of education status and trends; and report timely, useful, and high quality 
data to the U.S. Department of Education, the Congress, the states, other education policymakers, 
practitioners, data users, and the general public. 

We strive to make our products available in a variety of formats and in language that is appropriate to a 
variety of audiences. You, as our customer, are the best judge of our success in communicating information 
effectively. If you have any comments or suggestions about this or any other NCES product or report, we 
would like to hear from you. Please direct your comments to: 

National Center for Education Statistics 
Office of Educational Research and Improvement 
U.S. Department of Education 
555 New Jersey Avenue, NW 
Washington, DC 20208 



The NCES World Wide Web Home Page is 
http://nces. ed.gov 



Suggested Citation 

U.S. Department of Education. National Center for Education Statistics. Using Telephone and Mail 
Surveys as a Supplement or Alternative to Door-to-Door Surveys in the Assessment of Adult Literacy , 
Working Paper No. 2000-06, by Thomas G. Sticht. Project Officer, Sheida White. Washington, DC: 
2000. 



March 2000 



Foreword 



In addition to official NCES publications, NCES staff and individuals commissioned by 
NCES produce preliminary research reports that include analyses of survey results, and 
presentations of technical, methodological, and statistical evaluation issues. 

The Working Paper Series was initiated to promote the sharing of the valuable work 
experience and knowledge reflected in these preliminary reports. These reports are viewed as 
works in progress, and have not undergone a rigorous review for consistency with NCES 
Statistical Standards prior to inclusion in the Working Paper Series. 

Copies of Working Papers can be downloaded as pdf files from the NCES Electronic 
Catalog (http://nces.ed.gov/pubsearch/), or contact Sheilah Jupiter at (202) 502-7444, 
e-mail: sheilahjupiter@ed.gov, or mail: U.S. Department of Education, Office of Educational 
Research and Improvement, National Center for Education Statistics, 1990 K Street, NW, Room 
9048, Washington, DC 20006. 



Marilyn M. McMillen 
Chief Mathematical Statistician 
Statistical Standards Program 



Ralph Lee 

Mathematical Statistician 
Statistical Standards Program 




iii 



5 



Using Telephone and Mail Surveys as a Supplement or Alternative 
to Door-to-Door Surveys in the Assessment of Adult Literacy 



Prepared by: 

Thomas G. Sticht 

Applied Behavioral and Cognitive Sciences, Inc. 



Prepared for: 

U.S. Department of Education 
Office of Educational Research and Improvement 
National Center for Education Statistics 



March 2000 



This project was an activity of the Education Statistics Services Institute. 



Table of Contents 



Foreword iii 

Abstract viii 

Introduction 1 

The Search for Lower Cost Methods for Representing Adult Literacy 1 

Purpose and Plan of the Present Paper 2 

Part 1 . Issues of Validity in Adult Literacy Assessment 4 

Some Issues of Validity in the NALS and IALS 6 

Part 2. Review of Related Research 9 

Assessing Specific and General - ’g Cognitive Abilities Using Telephone and Mail Surveys ..9 
Testing Word Knowledge by Telephone to Estimate General Cognitive Aptitude Using 

Computer Adaptive Testing 10 

Assessing Adult Literacy-Based Knowledge and Literacy Practices by Telephone 11 

Part 3. A Cognitive Theory of Literacy 16 

The Human Cognitive System 1 7 

A Developmental Model of Literacy 19 

Literacy and the Merging of Speech and Graphics Technologies 23 

Major Features of Graphics Technology 25 

The Graphic Representation of Spoken Language 26 

Part 4. Issues in the Use of Telephone and Mail Surveys in Assessing Adult Literacy 28 

Sampling and Other Issues 33 

References 35 

Appendix 41 




vii 



7 



Abstract 



This paper discusses the use of telephone surveys as a substitute for, or addition to, the use of door-to-door 
surveys in assessing adult literacy ability. Part 1 addresses issues of validity in adult literacy assessment. Part 2 is a 
review of related research that has used telephone and mail surveys to assess the cognitive abilities of adults. 

Part 3 discusses the theoretical model of literacy that the National Adult Literacy Survey (NALS) and International 
Adult Literacy Survey (IALS) have used and how that model can be modified to incorporate a developmental and 
componential model of literacy that also includes the oral language aspects of literacy given in the Adult Education 
and Family Literacy Act of 1998. Finally, Part 4 addresses several issues about the use of telephone and mail 
surveys, including concerns about using these survey methods in such a way as to make continuity in trend data from 
previous surveys of adult literacy possible. 
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Introduction 



National assessments of adult literacy have been valued by adult educators because they serve the generally 
useful purpose of keeping the issue of adult literacy visible to policymakers, business and industry leaders, labor 
unions, educators and adults themselves, who may be stimulated to seek additional literacy development (Fagan 
1994). 

Recently, however, the assessment of adult literacy in industrial nations has taken on a new significance 
(The Organization for Economic Co-operation and Development (OECD), 1995). The OECD notes that, in regard to 
concerns for economic competitiveness among member nations, "... one area that is receiving growing attention from 
educational policymakers and analysts in a number of OECD countries is the direct measurement of literacy levels in 
the labor force of industrialized countries” (Benton andNoyelle 1992, 11). Indeed, results of the first International 
Adult Literacy Survey (IALS) involving some ten member nations of the OECD have recently been reported (OECD 
1995, 1997). 

Despite their usefulness to adult educators and policymakers, the cost involved in conducting adult literacy 
surveys is a matter of some concern to these same groups. In the United States, the National Adult Literacy Survey 
(NALS) (Kirsch, Jungeblut, Jenkins, and Kolstad 1993), conducted by the National Center for Education Statistics 
(NCES) of the U.S. Department of Education cost over $14 million for the national and state samples involved (A. 
Kolstad — personal communication, April, 1994). When asked at a planning meeting for the National Assessment .of 
Adult Literacy (NAAL), state directors of adult literacy education who were present reported that “...it would be 
more difficult to find the funds for another round” (U.S. Department of Education 1998, 5). 

The Search for Lower Cost Methods for Representing Adult Literacy 

The NALS of 1992 assessed adult literacy by personal interviews using door-to-door sampling. Interviewers 
asked adults to perform tasks such as filling in a bank check, locating times for travel on a train schedule, calculating 
the cost of a meal in a restaurant, and other types of “tasks that simulate the literacy demands that adults encounter in 
their daily lives” (Campbell, Kirsch, and Kolstad 1992, 12). 
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To avoid the costs of door-to-door assessments, the U.S. Department of Education has also used statistical 
regression techniques, based on data from the National Adult Literacy Survey, to estimate the literacy levels of adults 
in states that did not participate in the national survey (Reder 1994(a)). In Canada, too, concerns about the cost of 
administering direct assessments of adult literacy led the Department of the Secretary of State in Ottawa to explore 
the use of proxy measures of adult functional literacy to predict the direct assessment data (Neice, Adsett, and 
Rodney 1992). 

Statistical techniques and proxies for literacy assessment are of limited educational use when they include 
variables such as ethnicity and gender (Reder 1994(a)) or learning disabilities and mother’s educational level (Neice, 
Adsett, and Rodney 1992) in estimating people’s literacy skills. These variables are not amenable to change by 
education or training and hence recommendations for action in the form of educational interventions are not feasible. 

In a planning meeting for the NAAL, to be conducted in 2002, it was suggested that telephone interviews 
might be considered as a cost-reducing method for characterizing aspects of adult literacy (U.S. Department of 
Education 1998, 9, 17). Presently, the National Center for Education Statistics conducts biennial telephone surveys 
of adults to determine their participation in adult education, the types of education they engage in, and so forth. So 
the requisite sampling and telephone survey methodologies are familiar to NCES and it is well recognized that 
telephone survey methods are much less costly than door-to-door surveys (Chandler 1994). However, there was 
disagreement at the NAAL planning meeting about whether telephone interviews are a viable means for identifying 
whether people can “read, write, and comprehend” (17). 

Purpose and Plan of the Present Paper 

One outcome of the planning meetings for the 2002 NAAL was the suggestion that a paper be 
commissioned to discuss the feasibility of using telephone surveys as a substitute for or addition to the use of door- 
to-door surveys in assessing adult literacy ability (U.S. Department of Education 1998, 9). That is the primary 
purpose of the present paper. 

The paper includes the consideration of the use of mail surveys along with telephone surveys as methods 
that are lower in cost than door-to-door interviewing for assessing adult literacy abilities. 
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Part 1. Issues of Validity in Adult Literacy Assessment. The first part of the paper addresses issues of validity 
that have arisen in discussions of national assessments of literacy. This includes a discussion of construct or 
inferential validity and the role of convergent and discriminant evidence in establishing the validity of a theory of 
literacy. It also includes a discussion of the concept of action or use validity (Messick 1989). This discussion of 
validity sets the stage for a later discussion in Part 4 of the paper, which includes a comparison between a telephone 
survey approach and the NALS door-to-door approach in producing informational products deemed useful by 
policymakers who have supported these large-scale assessments. 



Part 2. Review of Related Research. The second segment of the paper provides a review of published studies that 
have used telephone and mail surveys to assess the cognitive abilities of children or adults. The research reviewed 
includes the assessment of “intelligence” (“g”) in studies of behavioral genetics, verbal “aptitude” in studies of 
personnel testing for selection of qualified applicants for military service, and two studies of adult literacy using 
telephone and mail surveys. 

Part 3. A Cognitive Theory of Literacy. The third section of the paper includes a discussion of the theoretical 
model of literacy that the NALS and International Adult Literacy Survey (IALS) have used. It discusses how that 
model can be modified to incorporate a developmental and componential model of literacy that also includes the oral 
language aspects of literacy given in the Adult Education and Family Literacy Act (AEFLA) of 1998: 

The term ‘literacy’ means an individual’s ability to read, write, and speak in English, compute, and 
solve problems, at levels of proficiency necessary to function on the job, in the family of the 
individual, and in society (page H6636). 

To speak in English also implies the ability to listen to and comprehend English. The importance of these 
oral language aspects of literacy (i.e., speaking and listening to comprehend the English language) to the learning 
and use of the written English language by native and foreign speakers alike is emphasized in the AEFLA, which 
stresses the importance of phonemic awareness and phonics in the literacy development of adults in six different 
sections of the law. The purpose of phonemic awareness and phonics training is to make it possible for a person to 
comprehend the written representation of the English language through reading with the same or greater 
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effectiveness and efficiency as the oral representation of the English language can be comprehended by listening. 

The theoretical model of literacy developed in Part 3 incorporates these various oral and written language constructs. 

Part 4. Issues in the Use of Telephone and Mail Surveys in Assessing Adult Literacy. The final part of the 
paper addresses several issues concerning the use of telephone and mail surveys, including concerns about obtaining 
representative samples and about using these survey methods in such a way as to make continuity in trend data from 
previous surveys of adult literacy possible. The feasibility of continuity in such NALS and IALS categories of 
information as literacy practices, intergenerational transfer, economic indicators, civic participation, and various 
demographic information products by telephone/mail surveys is explored. 

Part 1. Issues of Validity in Adult Literacy Assessment 



How should the literacy abilities of adults be represented? When discussed as a form of human cognitive 
ability, as in the NALS and IALS, literacy is a psychological construct and as such “levels” of literacy cannot be 
“directly measured” as suggested might be done in the 1992 report from the OECD. In fact, there are no “literacy 
levels” to be directly measured in the adult labor forces of nations. Instead, there are various ways of construing the 
nature of literacy and different procedures of measurement that can lead to the construction of alternative 
representations of adult literacy in society. 

In the United States over the last seventy-five years, different representations of adult literacy have been 
socially constructed by psychometricians, statisticians, and survey experts in consultation with various stakeholder 
groups including adult educators, adult students, literacy researchers, policymakers and others with a declared 
interest in adult literacy (Sticht and Armstrong 1994). For instance, the National Assessment of Educational Progress 
of 1970-71 assessed adult literacy using the same “academic” tasks that were used with school children, such as 
knowledge of word meanings (vocabulary), using visual aids, following written directions, using reference materials, 
locating significant facts, getting the main idea from materials, drawing inferences, and critical reading. 

The Young Adult Literacy Survey (YALS) of 1985, the NALS of 1992 and the IALS of 1995 are the most 
recent representations of adult literacy at national levels. These surveys assessed adult literacy using “tasks that 
simulate the literacy demands that adults encounter in their daily lives” (Campbell, Kirsch, and Kolstad 1992, 12). 
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Similar “real world” tasks were used earlier in the “Survival Literacy” survey by Louis Harris Associates in 1970, the 
Adult Functional Reading Study of 1973 and the Adult Performance Level Study of 1975 (Sticht and Armstrong 
1994). 

The fact that assessments of adult literacy over the decades have constructed various representations of 
adult literacy (“academic,” “real world”) raises important questions. Just how should the literacy abilities of adults be 
represented? Are all representations equally valid? Such questions seem to underlie the concern among participants 
at the NAAL planning meeting mentioned earlier on whether telephone interviews are a viable means for identifying 
whether people can “read, write, and comprehend.” 

Yet, even setting aside the concerns about whether telephone surveys can validly represent the literacy 
abilities of adults, the validity of the general practice of governments in conducting large scale assessments of 
people’s literacy skills has not gone unchallenged. For instance, Kazemek ( 1 990) stated: 



...attempts at defining “ability levels” and “norms” are not only futile but potentially 
dangerous as well. How can we possibly arrive at acceptable definitions of literacy when there are 
countless life goals, needs, and desires among the adult populations? Realistically we cannot, but 
in our attempts to do so we usually produce reductive lists, scales, and criteria... which are then 
used to categorize large segments of the population, often in detrimental ways. (56). (See also 
papers in Venezky, Wagner, and Ciliberti 1990). 



Kazemek’s questioning of the validity of large-scale assessments of adult literacy in terms of both the way 
one defines and measures literacy and the way one uses the scores from such assessments raises concerns addressed 
by Messick’s discussions of validity (1989, 1988). Messick discusses validity as a “unified concept” with different 
facets in which the validity of an assessment refers to the validity of the inferences one makes from the data. 

One facet of the validity of test scores is the traditional concept of construct validity. Messick refers to this 
as the validity of interpretive inferences and states: 



To validate an interpretative inference [italics added] is to ascertain the extent to which multiple 
lines of evidence are consonant with the inference, while establishing that alternative inferences are 
less well supported. This represents the fundamental principle that both convergent and 
discriminant evidence are required in test validation (1989, 5). 

Evidence of convergent validity arises when correlations of literacy scores are found with other variables (e.g., years 
of education, amount of reading) and when these relationships are predicted by the theory of literacy underlying an 
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assessment. Evidence of discriminant validity arises when variables that the theory of literacy predicts should not be 
related are, in fact, unrelated in the data. 

Kazemek’s questioning of the definitions and operations used to assess adult literacy is a challenge to the 
construct or interpretative validity of the assessments. His questioning of the uses of test scores, including 
detrimental uses, is addressed by Messick in the facet of validity that he calls action or use inferences. He states, “To 
validate an action inference [ italics added] requires validation not only of score meaning but also of value 
implications and action outcomes, especially of the relevance and utility of the test scores for particular applied 
purposes and of the social consequences of using the scores for applied decision making.” (1989, 5). 

In the present paper, evidence for action validity is evaluated in Part 4 through the comparison of telephone 
and door-to-door surveys as a means for producing information products about adult literacy valued by the U.S. 
Congress, the branch of government that requested the NALS in federal legislation. 

Some Issues of Validity in the NALS and IALS 

In the United States, despite strongly held doubts by some scholars regarding the feasibility of defining and 
assessing adult literacy, as indicated above, the U.S. Congress passed the Adult Education Amendments of 1988 that 
required the U.S. Department of Education to submit a report to Congress on the definition of literacy and then to 
report on the nature and extent of literacy among adults in the nation (Campbell, Kirsch, and Kolstad 1992). 

With the aid of a national Advisory Board, the NCES agreed to define literacy as “Using printed and written 
information to function in society, to achieve one’s goals, and to develop one’s knowledge and potential.” (Kirsch, 
Jungeblut, Jenkins, and Kolstad 1993, 2-3). 

The Advisory Panel for the NALS also agreed on the definitions of three different literacy scales that were 
developed to represent the literacy skills of adults. These included: prose literacy — the knowledge and skills needed 
to understand and use information from texts; document literacy — the knowledge and skills required to locate and 
use information contained in materials; and quantitative literacy — the knowledge and skills required to apply 
arithmetic operations embedded in printed materials (Kirsch, Jungeblut, Jenkins, and Kolstad 1993). These same 
definitions were used to assess adults’ literacy abilities in the IALS (OECD 1 995). 
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It should be noted that in both the NALS and the IALS there was an important conceptual shift in going 
from the general definition of literacy as “using printed and written information to function in society, etc.” to the 
definition of the three literacy scales that were actually going to be used to characterize adult literacy in the various 
countries. None of the definitions of the three scales started with “using printed and written information” as the 
defining aspect of literacy. Rather all three definitions referred to the psychological constructs of “the knowledge and 
skills” or “the ability” required to perform tasks using prose, document, and quantitative materials. This makes an 
important shift in conceptual understanding that, unlike what the general definition of literacy stated, literacy is not 
the use of printed materials per se, but rather the knowledge and skills or ability that make possible the use of printed 
materials for various purposes. 

A recent U.S. General Accounting Office reported that one expert on adult literacy research stated that 
functional literacy tests like those of the NALS and IALS may lack validity because they are not derived from 
theoretical models of ability but from everyday literacy tasks (U.S. General Accounting Office 1995, 24). Given the 
complex nature of such tasks, it is not clear what implications can be drawn from the test performance. 

This was clearly illustrated in a manual for item writing produced by the Comprehensive Adult Student 
Assessment System (CASAS), an adult literacy assessment system widely used in the United States and in some 
other industrialized nations that uses “real world” literacy tasks much like those of the NALS and IALS. The manual 
notes that the use of complex, “real world” or “functional” tasks as items “...generally tests the use of two or more 
skills [italics added]. Therefore, this context is not appropriate in itself for diagnosing weaknesses in specific skills 
since it is difficult to determine which skill was performed incorrectly” (CASAS 1983, 1). 

In the absence of a clearly specified theory of “literacy” as a psychological construct, it is not possible to 
know how to develop assessments that measure the component knowledge and skills that make-up the ability or 
abilities that constitute “literacy.” Without knowing what specific knowledge or skills are being assessed in “real 
world” tasks, it is not clear to what extent test performance reflects literacy ability or some other abilities, such as 
problem solving, reasoning, language comprehension, vocabulary knowledge, management of test-taking anxiety, 
interpersonal skills, or some complex, interactive combination of all these. That is why, in the earlier version of the 
NALS and IALS, referred to here as the YALS, performance on the three literacy scales was referred to as “complex 
information processing” tasks (Kirsch and Jungeblut 1986). 
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This problem of interpreting what is being measured may not be so important for those who perform well, 
and can therefore be assumed to possess whatever knowledge and skill is called for in performing the “complex 
information processing'’ (literacy?) tasks. The problem becomes critical when the focus of concern is on 
understanding why it is that those who do not perform well do not perform well. What kinds of services should be 
provided to help them improve their ability to perform these kinds of tasks? Should major efforts go into providing 
basic phonemic awareness or phonics training, or should vocabulary development and reading comprehension be the 
focus of instruction? Should there simply be practice in performing a wide variety of complex, information 
processing tasks like those assessed on the literacy surveys? How many of such tasks should be used in training and 
how and why would specific task training be expected to generalize to other tasks? Would training in memory 
enhancing techniques, logical reasoning, and critical thinking be the thing to do? How much of the problem is a 
“literacy” problem in contrast to a “general education” problem? 

The lack of a good, workable theory of literacy has led to the development of adult literacy assessments 
over the last three decades using a variety of different materials and tasks. Most recently the NALS and IALS have 
used three different but highly correlated scales (around r = .90, (Reder 1994(a); Rock 1998)) that produce much 
redundant information. If a theory of literacy could help identify the “knowledge and skills” that underlie the 
performance of prose, document and quantitative tasks it might be possible to more directly and cost-effectively 
assess that knowledge and those skills and to predict the performance of a wide variety of both “academic” and “real 
world” literacy tasks that go beyond those used in present assessments. 

While it is not possible at the present time to provide a comprehensive theory of literacy that is universally 
agreed upon by all concerned, Part 3 of this paper provides aspects of a cognitive theory of literacy that suggests 
some of the major components involved in literacy that might be assessed not only in door-to-door surveys, but also 
in telephone and mail surveys. 
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Part 2. Review of Related Research 



Cognition ; Cognitive (adjective): (1) The mental process or faculty by which knowledge is 
acquired; (2) that which comes to be known , as through perception , reasoning , or intuition; knowledge. 
(The American Heritage Dictionary 1976) 

The definition given above indicates that cognition includes (at least) two aspects: (1) the mental processes 
by means of which knowledge is acquired and (2) the knowledge that has been acquired. Telephone surveys have 
been used to obtain information about both aspects of cognition, the mental processes that people use and the 
knowledge that people possess. Three such studies are reviewed here. 

Assessing Specific and General - Cognitive Abilities Using Telephone and Mail Surveys 

In studies of behavioral genetics, Plomin and associates (Kent and Plomin 1987; Plomin et al. 1986) used 
mail and telephone surveys to assess various cognitive abilities of children, adolescents and adults (ages 9 to 80). 
Across the three studies, the tests assessed various verbal, spatial, perceptual speed, and memory abilities. The 
general procedure in three different studies (two reported in Plomin, et al. 1986) was to first mail out kits containing 
cognitive test batteries to individuals who had agreed to participate in the studies. Then the test administrator would 
telephone the subjects and administer the tests over the telephone. 

To check for comparability between telephone and face-to-face testing, Kent and Plomin (1987) 
administered face-to-face assessments to samples of the two hundred and twelve children and adolescents in their 
study. Fourteen were tested face-to-face before the telephone assessment and sixteen after the telephone tests. 
Additionally, thirty-two individuals had been tested earlier in telephone sessions and they provided test-retest 
information for the telephone assessments. Altogether then, there were sixty-two subjects with test-retest scores. 

Data showed correlations among telephone and face-to-face samples of over .92 for a g (general ability) 
factor extracted from the factor analysis of scores. Overall, for the total of 62 test-retest scores, correlations were .81 
for the Verbal factor, .73 for Spatial, .82 for Perceptual Speed, .57 for Memory, and .79 forg. 
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Kent and Plomin (1987) state that, “These studies indicated that it is possible to assess cognitive functioning 
via telephone; importantly, subjects indicated that they preferred telephone testing to face-to-face testing, citing 
increased anxiety during in-person testing” (392). 

Testing Word Knowledge by Telephone to Estimate General Cognitive Aptitude Using Computer 
Adaptive Testing 

Research by Legree, Fischl, Gade and Wilson (1998) aimed to determine whether word knowledge 
(vocabulary) could be assessed over the telephone to predict scores on the Armed Forces Qualification Tests 
(AFQT), a paper and pencil battery of tests used to select applicants for military service. 

Earlier research had produced the Computerized Adaptive Screening Test (CAST) to help U.S. Army 
recruiters quickly estimate the cognitive aptitude of potential recruits so they could concentrate their time on those 
individuals most likely to qualify for service. The CAST is highly correlated (.91 ) with the AFQT and requires about 
1 0 to 15 minutes for administration. This contrasts with the approximately three hours needed to take the Armed 
Services Vocational Aptitude Battery (ASVAB) from which the AFQT is derived using four sub-tests of the battery. 

The CAST uses a computerized adaptive testing program to select items from a large bank of items on the 
basis of the individual's current level of performance. Using this program, performance on the AFQT is estimated 
with approximately 10 items from the verbal (word knowledge) domain and 5 from the quantitative domain. 

To determine whether the CAST verbal items could be given by telephone to yield an estimate of 
psychometric g (general ability) and AFQT scores, the CAST verbal items consisting of multiple-choice, word 
knowledge questions requiring recognition of synonyms was administered over the telephone to 144 recruits who had 
recently enlisted but had not yet entered into the military. The use of new recruits meant that their paper-and-pencil 
ASVAB/AFQT scores were available from their official records and could be used to determine correlations among 
the telephone and CAST determined vocabulary knowledge scores and the AFQT scores. 

Subjects were called by experienced telephone interviewers who explained the purpose of the study and 
requested the subject's participation. The interviewer read each question and response alternatives (e.g., The word 
tall means: heavy, large, small, high). Subjects indicated their choice for each item and the interviewer entered the 
choice into a computer and the program then selected the next item. The branching algorithm selected 10 or 1 1 items 
for each subject in arriving at an estimate of the AFQT score. 
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Data analyses indicated that the telephone and CAST test scores correlated .66 with the AFQT. Because 
range restrictions due to military entrance requirements attenuates all ASVAB correlations, a bivariate correction for 
range restriction was used to estimate the correlation between the AFQT and the telephone CAST scores for the 
general population and this produced a correlation of .81. 

Based on the results of the study, the U.S. Army Research Institute is incorporating the Telephone Test into 
a larger research-based telephone survey addressing issues related to youth career selection to link individual 
differences in cognitive aptitude to other variables. 

The research will also explore the use of the Telephone Test with a broader range of abilities, with 
particular interest in lower aptitude groups. 

Assessing Adult Literacy-Based Knowledge and Literacy Practices by Telephone 

In a study using random-digit dialing telephone sampling techniques, Sticht, Hofstetter and Hofstetter 
(1996) assessed various aspects of adults’ literacy-based knowledge and use of printed materials at home and on the 
job. 



Defining Literacy-Based Knowledge. Several lines of research have converged suggesting that people become 

highly literate largely by engaging in numerous literacy practices, such as reading books, magazines, newspapers, 
and so forth (Reder 1994(b); Krashen 1993). A review of the major assessments of adult literacy in the United States 
since 1937 consistently revealed that as years of education increases there are corresponding increases in both the 
number of literacy practices in which adults engage and the amount of skill displayed in the assessments (Sticht and 
Armstrong 1994). 

In an illuminating program of research, Stanovich and associates explored how the extent of engagement in 
literacy practices by children and adults contributed to their development of literacy and, more broadly, “verbal 
intelligence” (Stanovich 1993; Stanovich and Cunningham 1993). Using an innovative method for assessing 
knowledge with checklists calling for a simple ‘‘yes” or “no” judgment on the part of the reader, Stanovich and 
associates demonstrated that performance on these checklists correlated significantly with a variety of literacy 
activities and cognitive assessments (Allen, Cipielewski, and Stanovich 1992; West, Stanovich, and Mitchell 1993). 
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A major conclusion from this body of research is that the simple checklists assess knowledge indicative of (1) the 
extent of use that one makes of printed and written materials, (2) the knowledge one derives from the use of these 
materials, and (3) the added potential for engaging in future literacy practices as a consequence of having a broader 
base of knowledge prerequisite for wide-ranging reading. 

In these studies, Stanovich and associates argue that individuals who read a lot acquire, mostly through 
incidental learning, a large declarative knowledge base containing the names of authors, magazines, newspapers, 
persons known for their contributions to film, theatre, music and other activities that Hirsch (1987) referred to as 
“cultural literacy” and a large vocabulary of words typically not encountered with high frequency in day-to-day oral 
communication nor on television or radio. Scores on these checklists are indicators of both the amount of reading in 
which individuals engage and of the cognitive outcomes in terms of the individual’s declarative knowledge base (see 
comments by Taylor 1994 and reply by Stanovich and West 1994, regarding contrasting perspectives on the use of 
the checklist methodology for studying reading behavior). 

Using the Stanovich Checklists to Assess Literacy-Based Knowledge by Listening on the Telephone . 

The checklist approach to knowledge assessment developed by Stanovich and associates lends itself readily to the 
assessment of knowledge by listening since, as suggested by the developmental model, below, research indicates 
that, beyond the period of reading in which early decoding skills are developed, listening and reading converge on 
the same knowledge base (Sticht et al. 1974; Sticht and James 1984; Stanovich 1991; Sinatra 1990). The theory that 
both listening and reading share the same knowledge base in the human cognitive system (see below), which is 
made-up of both language (grammar, with lexicon and syntax) and language-based meanings, means that in 
individuals, after reading decoding processes are developed, the processes of listening and reading are very largely 
the same at the level of the knowledge base (Sticht 1978; Sticht and McDonald 1992). In the individual’s mind, 
listening and reading processes are not correlated ; rather, they are structurally the same with regard to the knowledge 
base that they draw upon (with only slight differences due primarily to the nature of the informational displays 
involved in speaking and writing). 

Because individual differences exist in the amount of knowledge that individuals possess, listening and 
reading will tend to be moderately to highly correlated in the adult population, meaning that low, medium and highly 
knowledgeable adult listeners will also be low, medium and highly knowledgeable readers (Sticht, Hooke, and 
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Caylor 1981(a)). The correspondence of listening and reading processes at the level of the knowledge base forms the 
mechanism for assessing the knowledge component of literacy by telephone. 

Knowledge as an Indicator of Literacy Ability. There is a knowledge component (words, sentences, paragraphs) 
in all of the myriad literacy tasks in which people engage in our information age society. This suggests that the 
knowledge component can be used as an indicator of literacy ability. The assessment of knowledge as distinct from 
the performance of complex tasks that include knowledge along with various types of information processing tasks of 
unknown qualities is also warranted when the definition of literacy adopted by the advisory panel of experts for the 
NALS is examined. The definition of literacy agreed to was “Using printed and written information to function in 
society, to achieve one’s goals, and to develop one's knowledge [italics added] and potential” (Kirsch et al. 1993, 2- 
3). 

The importance of knowledge in literacy was also acknowledged by the NALS advisory panel in its 
acceptance of the definitions of the three different literacy scales (prose, document, quantitative) that were 
developed. In each case, literacy was defined as “the knowledge and skills” needed to perform the three types of 
literacy tasks. From these definitions, it is clear that the advisory panel for the NALS understood that the use of 
printed and written information to accomplish tasks requires certain knowledge and skills to make such use possible. 

In a telephone survey methodology, a respondent’s knowledge may be assessed using Stanovich-type lists 
containing discrete items, such as names or single vocabulary words, that require only a yes/no decision for each 
item. Such lists are suitable for presentation by telephone for listening and by written checklists for reading because 
they do not overload working memory (see the human cognitive system model, below) and introduce irrelevant task 
variance (Messick 1989). In contrast, the NALS used literacy tasks that emphasized “search and locate” (Guthrie, 
Britten, and Barker 1991) information processing skills requiring subjects to hold instructions in working memory 
while performing search and locate actions and problem solving (make inferences; reason) to accomplish unfamiliar 
tasks (Kirschet et al. 1993; Mosenthal and Kirsch 1991; Kirsch and Mosenthal 1990). 

NALS data for adults above age 54 show a rapid decline of performance (Kirsch et al. 1993). Since it is 
well established that working memory becomes increasingly less efficient with advanced age (Bernstein et al. 1 988; 
Meyer, Marsiske, and Willis 1993), these findings strongly suggest that the NALS and IALS tasks derive a great deal 
of their difficulty from the load they place upon working memory and this may threaten their validity as literacy 
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measures for the elderly. Additionally, the NALS methodology introduces unknown, and possibly irrelevant test 
variance which may lead to serious underestimations of the breadth of materials that older adults can read and 
comprehend using their knowledge base and the tasks they can perform in working memory given sufficient time to 
study materials, without the pressure for efficiency typical of test-taking situations. The latter are of questionable 
ecological validity in the lives of most adults over the age of 25 who are not in school. 

Questions for the present study. This study investigated the assessment of the knowledge component of adult 
literacy using telephone interviews with five hundred thirty-eight randomly selected adults, all English speakers, as 
the mode of presentation, listening as the mode of reception, and four abbreviated versions of the Stanovich 
checklists to assess aspects of the declarative knowledge base likely to be developed by reading. The checklists 
assessed adults’ knowledge of authors, magazine titles, famous people, and vocabulary. In addition to the telephone 
survey, a sub-sample of one hundred forty adults who agreed to complete a mailout version of the various checklists 
and a cloze test of reading was surveyed. Several questions that were explored are listed below along with the 
relevant findings. 

1. Does an oral presentation of checklists by telephone , which requires listening ability , produce 
results consistent with a written presentation of the checklists , which requires reading ability? 

Yes, data for the telephone and mailed surveys revealed positive relationships between listening and reading (r= 80 
for Total literacy scores made-up of the summed percentage correct across the four checklists). 

2. Do the knowledge scores obtained using the checklist and telephone survey methodology correlate 
with scores on another frequently used method for assessing reading ability? 

Yes, both the listening and reading checklist scores were positively and significantly (r=.38, p<.05) related to the 
cloze test with its greatly differing task demands and limited content (only one passage was used). These data, along 
with the data for the first question, provide evidence for the construct or interpretive validity (Messick 1989) of the 
theory of literacy that states that both listening and reading tap a common knowledge base and that the knowledge 
base is a component of, not a proxy for, literacy (reading, in this case). 
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3. Do the knowledge checklist scores from the telephone survey show positive relationships with 
demographic variables that have been found in previous adult literacy assessments? 

Yes, convergent evidence for interpretive validity was found in a positive manifold of statistically reliable correlation 
coefficients among the literacy knowledge checklists and other variables that are interpretable in relation to the 
development of literacy, such as parent’s education, one’s own years of education, amount of engagement in various 
reading practices, occupational status and income. 

4. Are the knowledge checklist scores from the telephone survey negatively correlated or uncorrelated 
with variables that the construct of literacy encompasses? 

Yes, discriminant validity was evidenced by data indicating that checklist scores were negatively correlated with age 
and the practices involved in family literacy (e.g., reading to children) which reflects the large proportion of older 
adults in the telephone sample whose children are grown and gone, yet whose literacy practices in other areas have 
tended to make them more literate (knowledgeable). 

This confirmatory evidence of construct validity for the checklist and telephone methodologies as measures 
of the knowledge component of literacy is encouraging given that this was the first study of its kind and no attempt 
was made to develop checklists or cloze tests that would provide greater differentiation among people. 

Domain Specific Knowledge Assessed by Telephone. In a second study of the assessment of adults’ knowledge 
by telephone, Hofstetter, Sticht and Hofstetter (1999) used six different checklists of knowledge, five that assessed 
adults’ domain specific knowledge of political information, such as famous political figures and various political 
policies, as well as a sixth checklist made up of the vocabulary checklist used in the earlier study. A random-digit 
dialing telephone survey of six hundred thirty-two English speaking adults provided data to study relationships 
among domain specific knowledge, general vocabulary knowledge, amount of reading reported, occupational status, 
income and participation in political activity. 

As in the first study, a positive matrix of significant correlations was found among literacy practices 
(amount of reading in home and work settings), domain and general knowledge, age, and amount of education. Of 
interest for demonstrating the importance of knowledge in the “information age,” positive and significant 
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correlations were found among knowledge and occupation, income and political activity even when data were 
statistically adjusted to control for age, education, gender, ethnicity, and measures of extent of literacy practice. 

The data from the two telephone surveys using the knowledge assessment checklist methodology indicate 
that a large amount of information about adults’ literacy-related knowledge can be obtained using telephone surveys. 
But can the information obtained by the telephone survey provide the same range of useful information that was 
obtained in the door-to-door NALS To answer this question, an action or use analysis (Messick 1989) of the five 
informational products that the NALS was commissioned to produce is presented in Part 4 to see how well the 
telephone data could be used to provide the same or related types of informational products. 

Part 3. A Cognitive Theory of Literacy 

Kirsch, Jungeblut, and Mosenthal 1998) trace the evolution of the theoretical framework used to construct 
national and international adult literacy surveys from the 1985 YALS, to the 1992 NALS and the 1995 IALS. Across 
these surveys, the framework used to develop assessment tasks was based on the types of materials involved in the 
assessments, the uses that respondents were asked to make of the materials, i.e., the types of tasks performed or 
questions to be answered, and what Kirsch, Jungeblut, and Mosenthal (1998) refer to as “...three important and 
distinct areas: prose, document, and quantitative literacy” (106). 

In a number of studies, it was found that difficulty of tasks (e.g., percentage getting various items correct) 
was a function of the information processing complexity of the task, including things like the numbers of items from 
the questions or task instructions that must be held in mind while searching through materials to locate answers, the 
amount of materials to be searched or read, or the abstract versus concrete nature of the language in the materials, or 
the types of and amounts of calculations that had to be done in the quantitative literacy tasks. 

But while there are a great deal of empirical data given to support the usefulness of the various material and 
task variables that the researchers identified through their analyses to predict the difficulty of prose, document and 
quantitative tasks, there is no explicit discussion of a cognitive theory of the competence (knowledge and skills) that 
adults must possess to be able to perform the many tasks. For instance, although all of the tasks required the use of 
reading decoding skills and language-based knowledge, and the prose, document and quantitative scales correlated 
around +.90, sharing some 80 percent of their variance, there is no theoretical statement about cognition that might 
account for this large overlap among the three literacy domains. 
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While theories of literacy acquisition vary, including sociolinguistic and sociocultural theories regarding the 
role of social interactions and cultural practices in the acquisition of literacy (e.g., Reder 1994(b)), the NALS and 
IALS have been developed based on a cognitive approach (e.g., the ‘‘knowledge and skills” needed for Prose, 
Document and Quantitative scales). Consistent with this conceptual stance, this section of the paper presents a 
cognitive theory of literacy that explains why there is so much overlap among the prose, document and quantitative 
scales and how it is possible to focus on different components of literacy to represent different aspects of adults’ 
performance of various literacy tasks. The theoretical framework includes a conception of (1) a simple model of the 
human cognitive system and (2) a conception of how that system develops from a prelanguage, preliterate system at 
birth into a language -based, literate system in adulthood. 

The Human Cognitive System 

The cognitive theory of adult literacy is based on the “modal model of memory” that has been examined in 
over thirty years of research (Figure 1) (Atkinson and Shiffrin 1971; Healy and McNamara 1996, 143). Generally 
speaking, highly literate individuals possess large bodies of knowledge in long term memory and information 
processing capacity and efficiency in working memory to process information in complex graphic documents 
(Kyllonen and Christal 1990). 

Cognitive development is construed as the change in the human cognitive system that results from changes 
in the anatomy and physiology of the human brain as it grows, develops, and deteriorates in later life. There are three 
concepts of “context” that are important in understanding cognitive development. Two kinds of context are “inside” 
the person. One is the biological, physiological context of the body and brain. The second is the mental context of 
the mind in which the person’s psychological life is constructed. The third concept of context refers to the world 
“outside” the person. This is the physical and social world in which the person lives, the external environment. 
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Figure 1. — Three components of the human cognitive system. 
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The internal environment or context, which is generally referred to as “mind,” is both the result of and the 
cause of cognitive development through experience. That aspect of mind involved in learning and cognitive 
development is referred to as the human cognitive system, and includes three major components of the cognitive 
system: the sensory-perceptual system, the short term (or working memory) and the long term memory (or 
knowledge base). The working memory is where active thinking takes place. Thinking processes lead us to pick up 
information from the outside environment and the internal knowledge base and combine these two sources of 
information to construct our understanding of the world at a particular moment. To a large extent, it appears to be the 
load or strain on working memory that accounts for much of the difficulty that Kirsch, Jungeblut, and Mosenthal 
(1998) related to the numbers of features that had to be held in mind and matched in the materials of the NALS or 
IALS, along with the types of calculations that had to be performed in working memory (“mental math”) in 
quantitative tasks. 

As the rough sketch below (Figure 2) indicates, a large amount of the overlap among the prose, document 
and quantitative tasks seems also to reflect the role of the knowledge base that is drawn on in comprehending the 
interviewer’s task instructions and in reading and comprehending the information in all three types of displays of 
information. As the figure illustrates, the vocabulary used to construct prose displays of information may also be 
used to construct document and quantitative displays. The extent of the vocabulary knowledge in adults’ knowledge 
bases no doubt contributed to their ability to deal with tasks involving “concrete” or “abstract” language and with 
larger amounts of information presented in written language. It should be noted that the commonality of language 
and working memory processes operate in most cognitive assessments, leading to what some have called “general 
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ability” or “psychometric “g” (Gottfredson 1997). The present cognitive theory goes beyond these concepts by 
emphasizing the three-part model of the human cognitive system and the multifaceted components of cognitive 
development given in the developmental model of literacy described below. 



Figure 2. — Illustration of how language-represented knowledge from the knowledge base of the human cognitive 
system model can be used across prose, document and quantitative literacy tasks to produce high correlations among 
the three literacy scales. 




A Developmental Model of Literacy 



A major goal of literate societies (and societies seeking literacy) is that each new generation should develop 
cognitive systems with the knowledge and processing skills needed to use (produce and access) the symbolic 
knowledge bases stored in the external environment in the form of books, maps, tables, and so forth. For this reason, 
a schematic representation of cognitive development in the literacy domain is used to organize information about the 
development of the human cognitive system as it transforms itself from preliterate to literate. 
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To (hopefully) facilitate understanding, the major concepts used in modeling the development of literacy in 
“the typical case of a child growing up in our literate society” are presented schematically in Figure 3. 

Before addressing the details of the model, several orienting comments regarding the figure are in order. 
First the figure is meant to portray a developmental sequence when examined from left to right. The sequence begins 
with the newborn infant (far left), and goes through stage 4 in which literacy skills are functional. The broad 
arrowhead on the far right is meant to imply continued development over the lifespan. The development of literacy, 
language, and knowledge is a lifetime activity. 



Figure 3. — A Developmental Model of Literacy 
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Examining the model from top to bottom, the top series of boxes is meant to represent the environment in 
which the person exists. This is the environment “outside the head.” This external environment makes available 
information displays that the person can explore and transform into internal representations of the external 
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information. These internal representations are developed by the processes in the second series of boxes labeled, on 
the far left, Information Processes in Working Memory. These processes go on “inside the head,” and merge 
information picked up from the external information displays with information picked up from the third series of 
boxes, labeled on the far left as Long Term Memory. Thus, the processes in the working memory are used to pick up 
and merge information from outside the body/brain with information in long term memory inside the body/brain to 
construct an internal representation of the world as currently experienced, including the construction of a meaning 
represented in symbolic information when this is a major domain of information being extracted from the external 
world at a given time. 

At the top of the model, there are references to four “stages.” In the present case, the term “stage” refers to 
what would typically be observed at different times if one studies children growing up in our literate society. For 
instance, stage 1 refers to the newborn infant who is considered to be innately endowed with the Basic Adaptive 
Processes involved in sensory/perceptual processes such as hearing and seeing, etc., motor movement, and cognition, 
including the processes needed to acquire information, mentally manipulate it, store it in memory, form knowledge 
structures out of it, retrieve and represent the information in various ways. In stage 1, these processes are assumed to 
work more or less automatically without conscious control, hence an observer would note that the infant seems 
“captured” by stimuli, rather than selective in observing information in the world. 

Stage 2 represents the emergence of conscious control over information pickup and manipulation. This 
active process of attending to information distinguishes listening from hearing, and looking from seeing, as 
information pick-up processes. Listening and looking build internal representations that may be called images. 
Images may also be constructed from data stored in long term memory. These internal imaging processes are 
frequently assessed in aptitude tests as “spatial perception” or “mechanical comprehension” in which it is necessary 
to mentally visualize and rotate cog-and-gear systems to determine what effects this movement might have on some 
other gear. 

Stage 2 also introduces the concept of active or working memory , which is defined by the occurrence of 
consciously controlled information processing activities. As noted earlier, working memory is a limited memory that 
can be easily overloaded (e.g., attending to two or three things at once is difficult — if not impossible). Many of the 
information processing activities the person acquires will be techniques to overcome active memory limits (e.g., 
repeating information to oneself keeps the information in active working memory until it can be applied). The 
“stage” aspect of cognitive development shows itself by the ability of the infant to attend to information selectively. 
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This is a cognitive capability, which, once developed, is a permanent feature of cognition that distinguishes the stage 
2 child from the stage 1 child. 

Stage 3 represents the development of language processes out of earlier processes and knowledge stored in 
long term memory. In developing oral language, the listening process is used in attending to spoken language to learn 
the words and grammar of language. Thus, listening plus languaging , occurs simultaneously. This joint occurrence is 
given the special name of auding. On the production side, the joint occurrence of uttering (making sounds through 
the mouth) with the production of word forms from the language pool, and stringing the word forms together to make 
sentences using the rules of syntax, produces the special process called speaking. Auding and speaking comprise the 
oral language information reception and production skills. Speaking is used to represent information that the person 
has in his or her mind “outside the head” and in the acoustic medium, while auding is used to pick up and transform 
speech information displays into knowledge in the mind of the listener. To an observer, the stage 3 child can respond 
to and produce oral language, at least to some minimal degree. Again, as with conscious attending, once oral 
language has begun to emerge, the cognitive system is permanently modified (barring physiological trauma of some 
sort), and the person is no longer capable of exclusively prelinguistic modes of thought. 

In transitioning to stage 4, the information processing skills of looking and marking are used to learn a 
representational system which, in many respects, represents the spoken language in a different medium — light — and 
in a more or less permanent graphic display: the written language. Looking at written language and transforming the 
written language into meaning is called reading. Writing is the special use of marking skills to produce graphic 
language (and other symbols and symbol systems). 

In the typical case, children develop a fair amount of competence in oral language before they are exposed 
to formal instruction in reading in elementary grades (though informal learning of literacy may begin in the home and 
community in literate cultures). In literacy instruction, written language skills build upon the earlier developed oral 
language skills and add to the person’s knowledge base special knowledge about how to represent and comprehend 
information in the graphic medium that was previously represented and comprehended by the person only in the oral 
language. Later, learning new vocabulary and conventions of language through reading and writing reflects back on 
oral language development and enlarges the person’s oral language abilities. The large arrow at the far right in 
Figure 3 is meant to represent the notion that the development of oral and written language ability may continue 
indefinitely as the person studies and develops new knowledge domains. 
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A major component of the developmental model of literacy is the person’s long term memory or 
“knowledge base.” The long term memory contains all the knowledge developed by the person in interaction with the 
environment. Much of the knowledge acquired by the person will not be understood in consciousness (for example, 
the rules of grammar). Rather, it will be unconsciously used to accomplish tasks such as developing language 
competency and comprehending the events of the world. In addition to the general world knowledge and processes 
that are in the mind, though not accessible to conscious understanding without considerable analysis, the memory 
also contains the language knowledge (words and grammar) that can be used to represent information that arises 
from experience in the world (e.g., bodies of knowledge about machines, parts of the body, houses, neighborhoods — 
sometimes called “schema” (Bransford et. al. 1986) or “mental models” (Rumelhart,1980), and from didactic 
instruction, as in training programs. 

The model holds that the development of the oracy skills of speaking and listening (auding) are built upon 
the prior development of prelinguistic knowledge through information processing activities. It is important that it be 
understood that this early, prelinguistic cognitive content, or knowledge, will form the foundation for the acquisition 
of new knowledge over the person’s lifetime, including that knowledge known as “literacy.” 

A final aspect of the model is that it recognizes that, on the one hand, the literacy skills of reading and 
writing utilize the same knowledge base that is used in auding and speaking, plus the special decoding and encoding 
skills of reading and writing. On the other hand, the very nature of the written language display — characterized by 
being more or less permanent, being arrayed in space, and utilizing the features of light (color, contrast) — makes 
possible the development of skills and knowledge entirely different from those involved in oral language. 

Literacy and the Merging of Speech and Graphics Technologies 

In all societies, preliterate people learn to communicate with others in their communities. One way they do 
this is by the use of visible movements called gestures. For example, they may make grimaces when food tastes bad 
to them, or frown when sad, or point to direct other’s attention to something. 

Another way preliterate people learn to communicate with others is through the development of a form of 
gesturing that transmits thoughts not by visible means, but by the invisible movement of the molecules of the sea of 
air in which people are immersed. That is, they acquire the oral language of their community. This means that their 
brains engage in a mental design and execution process that permits them to use their neck and chest muscles, lungs 



and oral apparatus (i.e., their articulators , tongue, throat, teeth, lips, larynx) and their ears and other auditory 
components to produce movements (speech gestures) that result in sounds and sequences of sounds that permit them 
to exchange thoughts with others. The brain engages in this mental process without the person being consciously 
aware of what the brain is doing or how it is doing it. That is, the actual mechanics of the acquisition of the spoken 
language is done automatically by the person’s brain. 

A child’s developmental process typically leads to the formulation of a mental (brain) system or component, 
called grammar , consisting of a lexicon (a vocabulary of forms, generally words, though sometimes word-parts, e.g., 
-ed, constructed by the mental design process that controls the articulators) and a syntax (rules for arranging the 
word forms into sentences) for conveying meaning through sounds in the air. From their parents, siblings and others 
in their social group children learn certain formalities or rules about how one talks about certain others, what forms 
of address to use, how to formulate and tell a narrative story about some event they experience and many other things 
about the use of their native language. 

The Person 's Use of the Graphic Medium for Cognition and Communication . In addition to using visual 
gestures and those involved in speech for communication, preliterate people also learn to use some forms of marking 
of surfaces as a visual, graphic medium of communication. For instance, they may tattoo their bodies with graphic 
designs having various symbolic meanings. They may weave such designs into blankets or clothes, carve designs on 
wooden utensils or paint them on pottery. 

Though the gesture movements that produce both spoken and graphic designs may be used for 
communication, the graphic medium has features that differ from speech which permit the user to accomplish 
cognitive and communication tasks that would not be possible otherwise due to the limits of human memory. These 
features include permanence , spatiality , and light. 

The relative permanence of graphic displays makes possible the storage of information “outside the head” 
and makes it necessary for others to learn to look at and interpret the graphic “gestures” if they wish to retrieve the 
stored information. Similarly, the fact that the graphics medium uses space and light makes possible the design of a 
wide variety of displays that permit preliterate people to perform a number of cognitive and communication tasks 
that are not possible with the temporally linear, fleeting spoken language. 
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Literacy as the Merging of Speech and Graphics Technologies. In anthropological terms, technology is the 
body of knowledge a civilization uses to fashion implements, extract or collect materials, or practice various arts and 
skills (Morris 1976). From this point of view, literacy may be thought of as technology for producing and 
comprehending graphic displays as tools (implements) for accomplishing various cognitive and communication 
activities (Harris 1986; Goody 1977). In writing, a person “extracts” knowledge from the brain and “collects” 

(stores) it in graphic displays. Then, through the practice of the skill of reading, the collected knowledge is extracted 
by the same or another person from the graphic display and reconstructed in the brain. 

Much of what is encountered in teaching and learning reading results from the fundamental characteristics 
of graphics displays. These characteristics permit literates to work with the graphics technology in certain ways. 
Furthermore, the products of the work of literates create new graphics displays (e.g., history timelines, TV schedules, 
troubleshooting flow diagrams, atlases). This imposes additional information processing demands upon new readers 
who wish to learn to work with these new displays. 

Major Features of Graphics Technology 

The major features of graphics technology, examples of the types of products that may be produced by 
those in command of the technology, and examples of the types of demands for information processing that the 
products of this technology may require include: 



(1) Permanence. Graphic information displays, such as this page of print, or forms, tables , graphs, and so forth, are 
more or less permanent. Therefore, they can be used to collect or store information, including an extended body of 
knowledge. The information can be stored over time and retrieved later on, and it can be transported across space. 

(2) Spatiality. Unlike speech, graphic information displays can be arrayed in space. Signs can be placed on doors, 
over buildings, alongside highways, and so forth; pages of print with words laid out spatially to permit the recreation 
of a temporal flow of speech can be constructed; forms can be developed with “slots” containing labels (“Name;” 
“Address;” etc., see Figure 2, above) and myriad other graphic tools to accomplish various information transmission 
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and processing tasks can be developed (Schwartz 1971), such as labels, lists, bus schedules; flow charts; tables; 
schematics; transparencies; and so on. 

(3) Light. The third major feature of graphics technology that is drawn on in literacy is the use of light. The marks 
that are made to produce such graphic symbols as written or printed words, numbers, arrowheads in procedural flow 
charts, the white space of the cells in a table (matrix) and so forth are constructed by structuring the light that leaves 
the surface of the graphic medium such that the eye can detect the structure in the display. 

In addition to aiding in overcoming memory limitations and facilitating learning through various semantic 
“chunking” and organizing devices, the properties of light are extensively used to aid attention during information 
processing. Bold print may be used to call attention to certain information and color can guide information 
processing, as when a red line is used in an electronics diagram to permit a particular circuit to be traced in an array 
of circuits printed in black. 

As with all technology, the power of graphics technology arises from its use to develop tools for amplifying 
and extending human capabilities (Bruner 1968). However, unlike hammers, sewing machines, automobiles, and 
other technologies that extend human strength, dexterity, or locomotion abilities, graphic technologies gain their 
power from their application to the extension of human cognition and the ability to manipulate information in 
symbolic form. 

In particular, the merging of graphics technology with spoken language, itself a form of human technology 
for communicating with symbols, produces the power behind, and the awe and appreciation of, literacy. 

The Graphic Representation of Spoken Language 

Perhaps the capstone achievement in graphics technology was the development of the alphabet , a relatively 
simple technology by means of which a few graphic marks can represent enough aspects of the oral language that the 
marks permit a reader to reconstruct an oral language-based message from the graphic display (McGuinness 1997). 
The importance of this is that it permits graphic language to draw on the power of oral language for representing and 
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communicating knowledge, while bringing the power of the three features of graphics technology to bear on the 
development of new knowledge and tools for thinking and problem solving. 

For those adults with little or no reading ability, instruction generally centers on decoding or reconstructing 
a spoken message from the graphic display or “code.” The written display is called a “code” because it is considered 
an alternative representation of speech. In a simple substitution code one element in a message, such as the sequence 
1 , 2, 3 is substituted by another set of symbols, such as a, b, and c. In decoding, then, a is converted to 1 , b to 2, and 
c to 3. 

Though such a simple, one-to-one correspondence of speech and graphic symbols does not hold over the 
full range of the English language, enough correspondence is there to make the technology work. In the teaching of 
decoding by the principle of the alphabet (Liberman, Shankweiler, and Liberman 1989) learners are taught to 
substitute a graphic alphabet symbol for a speech sound (phonemics; phonics). In teaching writing they are taught to 
make drawings (alphabet characters) that can, in turn, represent speech sounds. 

Discussion. The theoretical model of the human cognitive system given in Figure 1 along with the 
illustration of Figure 2 suggest that one reason that the prose, document and quantitative tasks of the YALS, NALS, 
and IALS are so highly correlated is because for any given adult, the item tasks in all three domains draw upon the 
same long term memory or knowledge base, the same language (vocabulary, syntax) system, the same working 
memory system, and the same sensory-perceptual system that the adults have developed over their lives as illustrated 
in Figure 3. 

As Figure 2 illustrates, it is possible to separate out the language component of the prose, document and 
quantitative items from the graphic display components of the tasks. The latter introduce much of the demand for 
visual search and locate processes that tend to place an increasing load on working memory and renders tasks more 
difficult. 

For many people, as characterized in the developmental model of literacy of Figure 3, reading and writing 
refers to one’s mastery of the alphabetic principle, that is, the idea that speech sounds can be represented in a graphic 
code called the alphabet and reading is the recoding of the graphic code into a spoken language-based meaning. In 
fact, the “simple view of reading” (Gough 1996) considers that R=D+C, Reading equals Decoding plus 
Comprehension, where the latter is measured by listening tests. Recent research has supported the “simple view of 
reading” (Chen and Vellutino 1997). 
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The developmental model, the “simple view of reading, “ and extensive empirical evidence (Sticht et al. 

1 974; Sticht, Hooke, and Caylor 1 98 1 (a); Sticht and James; 1 984; Sinatra; 1 990) suggest that once adults get beyond 
the decoding stage of reading, their comprehension of language can be assessed equally well by listening as by 
reading (see the Appendix to this paper). All of the studies of the use of telephone surveys demonstrated that they 
could assess vocabulary knowledge by listening, and these assessments correlated well (r’s in the +.80 range) with 
performance by reading in face-to-face or mail out assessments. 

But given that neither the actual reading of language nor the use of graphics displays occurred in the 
telephone studies reviewed above, the question remains as to whether or not the information obtained by telephone 
surveys can provide the same range of useful information that was obtained in the door-to-door NALS. This was the 
question posed earlier in this paper that addresses the action or use validity (Messick 1989) of the telephone 
methodology. The answer to this question is discussed next in the context of the analysis of the cost-effectiveness of 
the telephone survey methodology used by Sticht and associates (Sticht, Hofstetter, and Hofstetter 1 996; Hofstetter, 
Sticht and Hofstetter 1999) in contrast to door-to-door sampling for producing the range of informational products 
desired by the U.S. Congress when it commissioned the national survey. 



Regarding the cost of telephone surveys, in addition to sponsoring the collection of academic skill 
achievement data by direct testing in the United States, the National Center for Education Statistics (NCES) has an 
office for conducting telephone surveys. According to Dr. Kathryn Chandler of the NCES, costs per interview by 
telephone are a fraction of the costs of conducting door-to-door interviews (K. Chandler — personal communication, 
April, 1994). Chandler estimated that a sample of six thousand telephone interviews with adults might easily be 
achieved for less than $1 million. This suggests that to interview a national representative sample of 1 3,600 adults, 
with an oversampling of black and Hispanic households, as was done in the 1992 NALS (Kirsch et al. 1993), the cost 
would be approximately $2 million or less. This compares to the approximately $1 0.85 million cost of obtaining the 
national sample for the NALS. 



Part 4. Issues in the Use of Telephone and Mail Surveys in 



Assessing Adult Literacy 
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The concept of cost-effectiveness requires that both the cost and the effectiveness of alternative methods to 
achieve the same or very similar outcomes be considered. There is no question that the telephone survey 
methodology is a lower cost approach to interviewing than door-to-door canvassing. Many may, however, question 
whether the telephone methodology yields inferences that are as valid (useful) for characterizing the literacy of the 
adult population as those derived from the actual performance of literacy tasks, as in the 1992 NALS. Fortunately, 
the report of the design of the NALS provides a list of outcomes, uses, or “benefits” that the NALS was intended to 
provide (Campbell, Kirsch, and Kolstad 1 992, pp. 2-3). This makes it possible to conduct an action or use validity 
analysis to determine if the same or comparable outcomes as were produced by the NALS were also obtained by the 
telephone survey approach of Sticht, Hofstetter, and Hofstetter (1996). 

The following discussion first states a given informational product that the NALS was intended to provide 
(Campbell, Kirsch, and Kolstad 1992). This is followed by a discussion of what the NALS and the telephone survey 
method provided for the informational product. The NCES survey design report stated that the NALS would 
accomplish the following: 

( 1 ) Describe the levels of literacy demonstrated by the total adult population as well as by adults 
comprising various subgroups , including those targeted as “at risk. ” 

The NALS developed three groups of tasks called prose, document and quantitative literacy, administered 
the tasks to samples of adults, and used the tasks to scale both the adults’ literacy proficiencies on each of the three 
scales and the difficulty levels of the tasks using Item Response Theory. The difficulty level of each task was defined 
as the level of literacy needed to have “... an 80 percent probability of correct response.” (Kirsch, et al. 1 993, 71). 
Next, subjects were assigned to one of five literacy levels based on their proficiency scores. 

How well do these procedures characterize the literacy skills of adults? First, the rationale for the decision 
to scale the adults’ literacy proficiency using a probability of .80 of being able to perform a given task, when the 
probability could have been set at 70, 60 or any other percent, was not discussed (Kirsch et al. 1993). As Kolstad has 
recently noted (Kolstad, 1 996; Kolstad, et al. 1 998), changes in this percentage clearly change the outcomes of the 
analyses. Second, the decision to assign people to five levels of literacy proficiency based on their being able to 
perform 80 percent of the average tasks at a given level means that any competence to perform at higher levels was 
not “credited” to the adults. For instance, people who scored on average in Level 1 could also perform 50 percent of 
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Level 2 tasks, 25 percent of the Level 3 tasks, 20 percent of the Level 4 and one in six of the Level 5 tasks. Given 
these findings, it is difficult to ascertain what the literacy levels of adults “really 1 ' are. 

The telephone survey methodology also permits people to be assigned to levels based on the knowledge 
component of literacy. Levels of literacy knowledge can be established using the same methods as in the NALS, or 
using means and standard deviations to establish level ranges. With this approach, the decisions regarding what 
probability figures for task performance should be used are eliminated. Further, other available test information 
based on normal curve statistics, such as the Armed Services Vocational Aptitude Battery (ASVAB), become 
relevant to interpreting the survey data. 

(2) Characterize the demonstrated literacy skills in terms of demographic and personal background 
information. 

Major demographic data collected by the NALS have also been collected in telephone surveys. Whether the 
NALS-type of graphic displays or the telephone knowledge assessments are used, representations of literacy 
proficiency were positively and consistently related to education, age, ethnicity, income, occupational status, father 
and mother's education,. extent of engagement in literacy practices such as newspaper and magazine reading, and 
self-ratings of reading and writing abilities. These same relationships have been reported for the last seventy-five 
years in every major mass assessment of adult “intelligence, 11 “aptitude, 11 or “literacy 11 (Sticht and Armstrong 1994). 
These types of demographic/ knowledge/practice data can be obtained at much less cost by telephone surveys. 

(3) Characterize the work force of the country with respect to demonstrated literacy skills and activities 
reported by individuals in various occupational categories. 

The door-to-door and the telephone surveys both characterize the literacy proficiencies and practices of 
adults in major occupational groups. Both types of surveys produce similar results: laborers are not as proficient as 
clerical workers who are not as proficient as managers and professionals. These findings have been consistently 
found in numerous adult cognitive abilities assessments since the introduction by the U.S. Army of mass testing 
during World War I (Sticht and Armstrong 1994). The telephone survey method collects occupational data much less 
expensively than the door-to-door survey method. 
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(4) Provide an increased understanding of the skills and knowledge associated with functioning in a 
technological society. 

Probably the most important question that the NALS researchers were asked to report on was, “Are the 
literacy skills of America’s adults adequate ... to ensure individual opportunities for all adults, to increase worker 
productivity, or to strengthen America’s competitiveness around the world?” (Kirsch et al. 1993, xviii). The NALS 
report answered the question as, “Because it is impossible to say precisely what literacy skills are essential for 
individuals to succeed in this or any other society, the results of the National Adult Literacy Survey provide no firm 
answers to such questions” (Kirsch et al. 1993, xviii). The authors discussed the relation of being in the lower 
literacy levels to one’s social standing as indicated by more limited occupational opportunities, income, and so forth. 
From this, readers were invited to make inferences about how lower literacy skills may tend to limit one’s 
functioning in society. But all of these same relationships are readily studied by the telephone survey methodology at 
a lower cost and with fewer decisions having little or no rationale presented. 

It might even be argued that the assessment of knowledge (by checklists and/or other methods) is a more 
useful method for characterizing the “skills and knowledge associated with functioning in a technological society.” 
Several studies have demonstrated that high levels of “prior” or “background” knowledge in a specific domain can 
compensate for several “years” of “general” reading skill (Recht and Leslie 1988; Sticht et al. 1986). The NALS 
study itself concluded that high levels of “prior” or “background” knowledge about what one reads is prerequisite for 
comprehending at a high level across the wide range of tasks in the battery. 

For these reasons, it seems likely that assessments of knowledge might better predict the performance of 
“real world” literacy tasks than the NALS-type scales. As illustrated by the research reviewed above, the Armed 
Services have spent decades and tens of millions of dollars on the Armed Services Vocational Aptitude Battery 
(ASVAB), consisting of ten tests, all of which require some reading and eight of which are tests primarily involving 
general and special vocabulary and conceptual knowledge (e.g., knowledge of geometry, electronics, automobiles, 
etc.) (Sticht and Armstrong 1994). A composite of four of these knowledge tests forms the Armed Forces 
Qualification Test (AFQT) used to select applicants for military service. As noted in Part 2 (above), a telephone 
survey methodology using just vocabulary (word knowledge) assessment was able to estimate the AFQT scores of 
personnel with just a 10 to 1 1 item test using an adaptive testing algorithm. Since the AFQT is used to determine 
ones qualification for military service, this supports the position that knowledge assessment can serve to identify 
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those who can use printed and written materials to function in society, at least in the high-technology world of the 
armed services. 

(5) Interpret the findings related to information-processing skills and strategies in a way that can inform 
curriculum decisions pertaining to the education and training of adults. 

In addition to wanting information about adults’ literacy skills, Congress wanted information regarding 
instructional “remedies” that might be taken to improve adults’ literacy skills. The developers of the NALS 
suggested that adult basic skills programs should be geared to improving adults’ skills in prose, document and 
quantitative (PDQ) literacy (Mosenthal and Kirsch 1994). Indeed, researchers at the Educational Testing Service 
worked on an interactive video, computer-based instructional series that would teach document literacy skills. A 
small pilot study with a group of some ten to twelve adult basic skills students indicated that, while students made 
improvements in document literacy, they made three to four times as much gains on prose and quantitative literacy 
tests as on the document literacy tests. This led the instructor who administered the pilot course to observe that, “The 
gains were interesting considering the PDQ curriculum did not include instruction in these skills.” (Orr-Holley 1992, 
1 ). 

Yet, as noted earlier, the human cognitive system model and the developmental model of literacy suggest 
that the three scales are not separate areas of competence. Rather, they draw upon the same underlying cognitive 
system with its knowledge base and working memory processes. For this reason, it is to be expected that many 
language and processes developed in one domain may be available to other domains because they all draw on the 
same cognitive system. The results of the pilot instructional program would seem consistent with this point of view. 

The research reviewed above adds support to Stanovich’s (1993) socially relevant work on the origins of 
“verbal intelligence.” The theory behind the knowledge-based approach to literacy assessment used in the telephone 
and checklist survey is the practice-engagement theory of literacy development (Reder 1994(b); Stanovich 1993; 
Krashen 1993). This theory holds that, by engaging in extensive practices involving a wide range of reading, literates 
build vast bodies of knowledge (both declarative and procedural) and automaticity of word recognition that in turn 
make it possible to engage in and successfully complete a large number of literacy tasks. This suggests one simple 
recommendation for curriculum development for adult literacy programs from the telephone and checklist 
methodology of Sticht, Hofstetter, and Hofstetter (1996) and Hofstetter, Sticht and Hofstetter, (1999). To help 
people develop large bodies of knowledge and hence to become highly literate, literacy programs should arrange 
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conditions that will encourage students to engage in extensive, wide-ranging, substantive listening and reading over 
long periods of time. This seems to be the sort of thing that participation in long term education does. 

Sampling and Other Issues 

The use of sampling techniques raises questions about how well telephone samples represent the adult 
population in general, and particular segments in particular. Some may question the validity of the inferences derived 
from the telephone methodology for accurately representing the literacy of adults because some adults do not have 
telephones, and they are likely to be the least educated for whom literacy is a major concern. But sampling issues 
plague all surveys, including the NALS and telephone surveys (Dillman 1978; Frey 1989). Typically, demographic 
data from the census can be obtained to permit quite accurate extrapolations of telephone estimates to households 
without telephones and for the least educated. In fact, as noted earlier, the U.S. National Center for Education 
Statistics already has an office that uses biennial telephone surveys to obtain information about adult education for 
national policy purposes. The same sampling methods can be used to acquire a representative sample of adults for 
literacy assessment (K. Chandler — personal communication, April, 1994). 

Additionally, special populations in correctional facilities or in adult literacy education programs might be 
sampled using both face-to-face, paper- and-pencil surveys and telephone surveys to establish correspondences 
among telephone knowledge- and self-rating-based estimates of literacy. 

Another issue is possible public resistance to the telephone methodology for assessing adult literacy because 
there is no actual reading or writing involved. In this regard, it should be noted that Albert Shanker, late President of 
the American Federation of Teachers union, devoted his entire Sunday New York Times column of January 14, 

1996, to a report on the importance of developing knowledge in the public schools based on the telephone study of 
Sticht, Hoffstetter, and Hoffstetter ( 1 996). This suggests a degree of popular acceptance of the telephone and 
checklist methodology for assessing literacy. 

Additionally, as the research reviewed indicates, mail surveys may be used along with telephone surveys to 
make it possible to assess adults’ use of written materials and gain acceptance of the telephone methodology for 
assessing literacy. It is not necessary to limit assessments to the use of the checklist methodology. To avoid the types 
of criticisms of “cultural imperialism” that faced Hirsch (1987), for instance, a kit of materials might be mailed to a 
sub-sample of persons contacted by telephone and then an 800 telephone number could be used for the kit recipient 
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to call in to have a set of directions for performing various tasks with the written materials. Or there could be a call- 
back time for the telephone test administrator to contact the kit recipient. For instance, if the kit included a small 
book, then the interviewer might call and ask the respondent to turn to a given page and read a specified sentence 
aloud or to summarize the main idea of a paragraph. The book could include stories or information representative of 
many different cultural groups and the same methods used by the NALS, IALS and the forthcoming NAAL for 
ensuring acceptance by various cultural groups could be used in the design of the book or kit of materials. 

If a page of the small book contained a train schedule, then the examiner might ask the respondent the same 
sort of working memory loaded tasks involved in search and locate questions used with the document items of the 
NALS. 

Additionally, reading-related assessments such as phonemic awareness knowledge might be assessed via 
telephone using phonemic blending or segmentation tests. With a mail- out kit, the adult might be asked to read 
aloud a list of pronounceable nonsense “words.” Such techniques as the “book comprehension” mail-out and 
telephone methodology coupled with assessments of decoding skills might be especially useful in generating new 
information about the instructional needs of adults in NALS/IALS/NAAL Levels 1 and 2. 

Of course, if used as a supplement to a decennial adult literacy door-to-door assessment, then a telephone 
and/or mail-out survey methodology and instruments could provide information that has been correlated with NALS- 
type items in addition to other, knowledge based items to track adults’ literacy during years in between the decennial 
assessments, perhaps as a part of the NCES regular biennial adult education surveys that are presently conducted. 
This would couple the telephone methodology with the NAAL methodology and permit the tracking of change over 
time using the NAAL approach every ten years and the telephone/mail approach in between and every ten years. 



This paper was prepared under a contract with the Education Statistics Service Institute (ESSI) with funding 
from the U.S. Department of Education, National Center for Education Statistics. The opinions and conclusions 
stated are those of the author and do not necessarily represent the opinions, policies or positions of the ESSI or the 
U.S. Department of Education, National Center for Education Statistics. 




34 



42 



References 



Allen, L., Cipielewski, J., and Stanovich, K. E. (1992). Multiple Indicators of Children’s Reading Habits and 

Attitudes: Construct Validity and Cognitive Correlates. Journal of Educational Psychology , 84: 489-503. 

Atkinson, R. C., and Shiffrin, R. M. (1971). The Control of Short-term Memory. Scientific American, 225: 82-90. 

Benton, L., and Noyelle, T. (Eds.) (1992). Adult Illiteracy and Economic Performance. Paris, France: Organisation 
for Economic Co-operation and Development, Centre for Educational Research and Innovation. 

Bernstein, D., Roy, E., Srull, T., and Wickens, C. (1988). Psychology. Boston, MA: Houghton Mifflin Company. 

Bransford, J., Sherwood, R., Vige, N., and Rieser, J. (1986). Teaching Thinking and Problem Solving. American 
Psychologist, 41: 1078-1089. 

Bruner, J. (1968). Processes of Cognitive Growth: Infancy. Worcester, MA: Clark University Press. 

Campbell, A., Kirsch, I. S., and Kolstad, A. (1992). Assessing Literacy: The Framework for the National Adult 
Literacy Survey (NCES 92-1 13). U.S. Department of Education. Washington, DC: U.S. Government 
Printing Office. 

CASAS (1983). The CASAS Manual for Item Writing. San Diego, CA: Comprehensive Adult Student Assessment 
System. 

Chen, R., and Vellutino, F. (1997). Prediction of Reading Ability: A Cross-Validation of the Simple View of 
Reading. Journal of Literacy Research, 29: 1-24. 

Dillman, D. A. (1978). Mail and Telephone Surveys: The Total Design Method. New York, NY: John Wiley and 
Sons. 

Fagan, W. T. (1994). Adult Literacy Surveys: A Trans-border Comparison. Journal of Reading, 38: 260-269. 

Frey, J. H. (1989). Survey Research by Telephone (2 nd ed.). Newbury Park, NY: Sage Publications. 

Goody, J. (1977). The Domestication of the Savage Mind. Cambridge, England: Cambridge University Press. 

Gough, P. (1996). How children learn to read and why they fail. Annals of Dyslexia, 46: 3-20. 

Gottfredson, L. (1997). Why g Matters: the Complexity of Everyday Life. Intelligence, 24: 79-132. 

Guthrie, J. T., Britten, T., and Barker, K. G. (1991). Role of Document Structure and Metacognitive Awareness in 
the Cognitive Process of Searching for Information. Reading Research Quarterly, 226: 300-324. 

Harris, R. (1986). The Origin of Writing. London, England: Duckworth. 




43 



35 



Healy, A. F., and McNamara, D. S. (1996). Verbal Learning and Memory: Does the Modal Model Still Work? In J. 
Spence, J. Darley, and D. Foss (Eds.), Annual Review of Psychology, 47: 143-172. Palo Alto, CA: Annual 
Reviews, Inc. 

Hirsch, Jr., E. D. (1987). Cultural Literacy: What Every American Needs to Know. Boston, MA: Houghton Mifflin 
Company. 

Hofstetter, R., Sticht, T., and Hofstetter, C. (1999). Knowledge, Literacy, and Power. Communication Research, 26: 
58-80. 

Kazemek, F. E. (1990). Adult Literacy Education: Heading into the 1990s. Adult Education Quarterly, 41: 53-62. 

Kent, J., and Plomin, R. (1987). Testing Specific Cognitive Abilities by Telephone and Mail. Intelligence, 11: 391— 
400. 

Kirsch, I., and Jungeblut, A. (1986). Literacy: Profiles of America *s Young Adults. Princeton, NJ: Educational 
Testing Service, National Assessment of Educational Progress. 

Kirsch, I. S., and Mosenthal, P. B. (1990). Exploring Document Literacy: Variables Underlying the Performance of 
Young Adults. Reading Research Quarterly, 25: 5-30. 

Kirsch, I., Jungeblut, A., and Mosenthal, P. (1998). Adult Literacy in OECD Countries: Technical Report on the 
First International Adult Literacy Survey (NCES 98-053). U.S. Department of Education. Washington, 

DC: U.S. Government Printing Office. 

Kirsch, I., Jungeblut, A., Jenkins, L., and Kolstad, A. (1993, September). Adult Literacy in America: A First Look at 
the Results of the National Adult Literacy Survey (NCES 93-275). U.S. Department of Education. 
Washington, DC: U.S. Government Printing Office. 

Kolstad, A. (1996). The Response Probability Convention Embedded in Reporting Prose Literacy Levels from the 
1992 National Adult Literacy Survey. U.S. Department of Education. Washington, DC: National Center for 
Education Statistics. 

Kolstad, A., Cohen, J., Baldi, S., Chan, T., DeFur, E., and Angeles. (1998). The Response Probability Convention 

Used in Reporting Data from IRT Assessment Scales: Should NCES Adopt a Standard ? U.S. Department of 
Education! Washington, DC: National Center for Education Statistics. 

Krashen, S. (1993). The Power of Reading: Insights from the Research. Englewood, CO: Libraries Unlimited. 

Kyllonen, P. C., and Christal, R. E. (1990). Reasoning Ability is (little more than) Working-memory Capacity?!. 
Intelligence, 14: 389-433. 




36 



44 



Legree, P., Fischl, M., Gade, P., and Wilson, M. (1998). Testing Word Knowledge by Telephone to Estimate 
General Cognitive Aptitude Using an Adaptive Test. Intelligence, 26: 91-98. 

Liberman, I., Shankweiler, D., and Liberman, A. (1989). The Alphabetic Principle and Learning to Read. In D. 

Shankweiler and I. Liberman (Eds.). Phonology and Reading Disability: Solving the Reading Puzzle. Ann 
Arbor, MI: The University of Michigan Press. 

McGuinness, D. (1997). Why Our Children Can't Read and What We Can Do About It. New York, NY: The Free 
Press. 

Messick, S. (1989). Meaning and Values in Test Validation: The Science and Ethics of Assessment. Educational 
Researcher, 18: 5-11. 

Messick, S. (1988). The Once and Future Issues of Validity: Assessing the Meaning and Consequences of 

Measurement. In H. Wainer and H. I. Braun (Eds.), Test Validity (pp. 33-45). Hillsdale, NJ: Erlbaum. 
Meyer, B. J. F., Marsiske, M, and Willis, S. L. (1993). Text Processing Variables Predict the Readability of 
Everyday Documents Read by Older Adults. Reading Research Quarterly, 28: 235-248. 

Morris, W. (Ed.). (1976). The American Heritage Dictionary of the English Language. Boston, MA: Houghton 
Mifflin Company. 

Mosenthal, P. B., and Kirsch, I. S. (1991). Toward an Exploratory Model of Document Literacy. Discourse 
Processes, 14: 147-180. 

Mosenthal, P. B., and Kirsch, I. S. (1994, December). Defining the Proficiency Standards of Adult Literacy in the 
U.S.: A Profile Approach. Paper presented at the annual meeting of the National Reading Conference. 
Syracuse, NY: Syracuse University. 

Neice, D., Adsett, M., and Rodney, W. (1992). Direct Versus Proxy Measures of Adult Functional Literacy: A 

4 

Preliminary Re-examination. In L. Benton and T. Noyelle (Eds.), Adult Illiteracy and Economic 
Performance (pp. 69-87). Paris, France: OECD, Centre for Educational Research and Innovation. 
Orr-Holley, L. (1992). Operation PDQ : An ETS Pilot Program. Jacksonville, FL: Florida Community College at 
Jacksonville, Urban Resource Center. 

Organization for Economic Co-operation and Development. (1995). Literacy, Economy, and Society. Paris, France: 
Organisation for Economic Co-operation and Development, Centre for Educational Research and 
Innovation. 




37 



45 



Organization for Economic Co-operation and Development. (1997). Literacy Skills for the Knowledge Society. Paris. 
France: Organisation for Economic Co-operation and Development. 

Plomin, R., McCleam, G., Pederson, N., and Nesselroade, J. (1986, June). Adopted and Non-adopted Twin 

Resemblance for Cognitive Abilities Assessed by Telephone Testing in the Swedish Adoption/Twin Study of 
Aging. Paper presented at the annual meeting of the Behavior Genentics Association, Honolulu, HI. 

Recht, D. R., and Leslie, L. (1988). Effect of Prior Knowledge on Good and Poor Reader’s Memory of Text. Journal 
of Educational Psychology, 80: 16—20. 

Reder, S. (1994a). Synthetic Estimates of National Adult Literacy Survey Literacy Proficiencies from 1990 Census 
Microdata. U.S. Office of Education, Office of Vocational and Adult Education, Division of Adult 
Education and Literacy. Washington, DC: U.S. Government Printing Office. 

Reder, S. (1994b). Practice-engagement Theory: A Sociocultural Approach to Literacy Across Languages and 
Cultures. In B. M. Ferdman, R-M. Weber, and A. G. Ramirez (Eds.), Literacy Across Languages and 
Cultures (pp. 33-74). New York, NY: State University of New York Press. 

Rock, D. (1998, January). Validity Generalization of the Assessment Across Countries. In Murray, T., Kirsch, I. and 
Jenkins, L. (Eds.), Adult literacy in OECD countries: Technical Report on the First International Adult 
Literacy Survey. Washington, DC: National Center for Education Statistics. 

Rumelhart, D. (1980). Schemata. In B. Bruce and W. Brewer (Eds.), Theoretical issues in Reading Comprehension. 
Hillsdale, NJ: Lawrence Earlbaum. 

Schwartz, S. (1971). Modes of Representation and Problem Solving: Well Evolved is Half Solved. Journal of 
Experimental Psychology, 2: 347-350. 

Sherman, R., Condelli, L., and Koloski, J. (1998). Developing the National Assessment of Adult Literacy: 

Recommendations from Stakeholders (NCES 98-17). U.S. Department of Education. Washington, DC: 
NCES Working Paper. 

Sinatra, G. M. (1990). Convergence of Listening and Reading Processing. Reading Research Quarterly, 25: 115- 
130. 

Stanovich, K. E. (1991). Discrepancy Definitions of Reading Disability: Has Intelligence Led Us Astray? Reading 
Research Quarterly, 26: 7-29. 

Stanovich, K. E. (1993). Does Reading Make you Smarter? Literacy and the Development of Verbal Intelligence. In 
H. Reese (Ed.), Advances in Child Development and Behavior (vol. 24). New York, NY : Academic Press. 



Stanovich, K. E., and West, R. F. (1994). Reply to Taylor. Reading Research Quarterly , 29: 290-291 . 

Stanovich, K. E., and Cunningham, A. E. (1993). Where Does Knowledge Come From? Specific Associations 

between Print Exposure and Information Acquisition. Journal of Educational Psychology , 85: 2 1 1-229. 

Sticht, T. G. (1978). The Acquisition of Literacy by Children and Adults. In F. Murray and J. Pikulsky (Eds.), The 
Acquisition of Reading (pp. 131-162). Baltimore, MD: University Park Press. 

Sticht, T. G., and Armstrong, W. B. (1994). Adult Literacy in the United States: A Compendium of Quantitative 
Data and Interpretive Comments. Washington, DC: National Institute for Literacy. 

Sticht, T. G., and James, J. H. (1984). Listening and Reading. In P. Pearson, R. Barr, M. Kamil, and P. Mosenthal 
(Eds.), Handbook of reading research (pp. 293-317). New York, NY: Longmans. 

Sticht, T. G., and McDonald, B. A. (1992). Teaching Adults to Read. In S. J. Samuels and A. E. Farstrup (Eds.), 
What Research Has to Say about Reading Instruction (pp. 314-334). Newark, DE: International Reading 
Association. 

Sticht, T. G., Hooke, L. R., and Caylor, J. S. (1981). Literacy, Oracy, and Vocational Aptitude as Predictors of 
Attrition and Promotion in the Armed Services ( HumRRO FR-ETSD-8 1-1 1). Alexandria, VA: Human 
Resources Research Organization. 

Sticht, T. G., Hooke, L. R., and Caylor, J. S. (1981). Manual for the Administration and Interpretation of the 
Literacy Assessment Battery (LAB) (HumRRO FR-ETSD-8 1-10). Alexandria, VA: Human Resources 
Research Organization. 

Sticht, T. G., Beck, L., Hauke, R., Kleiman, G., and James, J. (1974). Auding and Reading: A Developmental Model. 
Alexandria, VA: Human Resources Research Organization. 

Sticht, T. G., Armijo, L. A., Koffman, N., Roberson, K., Weitzman, R., Chang, F., and Moracco, J. (1986). 

Teachers, Books, Computers, and Peers: Integrated Communications Technologies for Adult Literacy 
Development. Monterey, CA: U.S. Naval Postgraduate School. 

Sticht, T. G., Hofstetter, C. R., and Hofstetter, H. H. (1996). Assessing Adult Literacy by Telephone. Journal of 
Literacy Research, 28: 525-559. 

Taylor, D. (1994). The Trivial Pursuit of Reading Psychology in the “Real World”: A Response to West, Stanovich, 
and Mitchell. Reading Research Quarterly, 29: 277-288. 

United States General Accounting Office (1995). Adult Education: Measuring Program Results Has Been 
Challenging (GAO/HEHS-95-153). Washington, DC: U.S. Government Printing Office. 



Venezky, R. L., Wagner, D. A., and Ciliberti, B. S. (Eds.) (1990). Toward Defining Literacy. Newark, NJ: 
International Reading Association. 

West, R. F., Stanovich, K. E., and Mitchell, H. R. (1993). Reading in the Real World and its Correlates. Reading 
Research Quarterly. 28, 35-50. 




40 



48 



Appendix 



This appendix summarizes an empirical study of the proposition stated on page 20 of the paper that “once adults get 
beyond the decoding stage of reading, their comprehension of language can be assessed equally well by listening as 
by reading.” 

The manual for the Literacy Assessment Battery (LAB) (Sticht, Hooke, and Caylor 1981(b)) includes data showing 
the results of testing five hundred ninety-three applicants for military service using both the Gates-MacGinitie 
Reading Test (Level D) and the Listening and Reading Paragraphs subtests of the LAB. The latter was developed for 
the U.S. Air Force to measure the ability of adults to comprehend language by listening and reading. The LAB 
includes other components, too, but they are not relevant to the present discussion. 

With test scores on the LAB Listening and LAB Reading subtests as well as the G-M reading test, made up of 
combined vocabulary and paragraph reading subtests, it was possible to develop regression formulas to estimate G- 
M reading scores from LAB Listening and Reading subtest scores. This permits one approach to determining the 
extent to which the above proposition is accurate. If it is assumed that the widely used G-M is a valid measure of 
language comprehension by reading, then the question is, does the LAB Listening subtest estimate G-M reading in a 
similar manner as the LAB Reading subtest? 

A maximum score of 24 points was possible on either the Listening or Reading LAB subtests. The figure shows 
estimated G-M reading grade levels as a function of raw scores on the LAB paragraph tests for Listening and 
Reading. 



The correlation of LAB 
Listening with Lab Reading was 
.749 (56% shared variance). 
LAB Reading with G-M 
Reading was .804 (65% shared 
variance), LAB Listening with 
G-M Reading was .744 (55% 
shared variance). 

By way of comparison, the 
G-M correlated with the Nelson 
Denny Reading test .696 (48% 
shared variance) for a sample of 
1,020 applicants for military 
service. Thus the LAB 
Listening subtest was more 
highly correlated with the G-M 
reading test than was the Nelson 
Denny reading test. 

The data from the LAB, G-M, 
and Nelson Denny testing are 
consistent with the proposition 
that “once adults get beyond the 
decoding stage of reading, their 
comprehension of language can 
be assessed equally well by 
listening as by reading.” 



Estimated 

G-M 

Reading 

Grade 

Level 
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95-13 Assessing Students with Disabilities and Limited English Proficiency 

97- 29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? 

97-30 ACT’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable 
Assessment Results 

97-3 1 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational 
Progress 

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background 
Questions) 

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items 

97 - 44 Development of a SASS 1 993-94 School-Level Student Achievement Subfile: Using 

State Assessments and State NAEP, Feasibility Study 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1 988 

Beginning students in postsecondary education 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

Civic participation 

97-25 1 996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 

Climate of schools 

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 
in NCES Surveys 

Cost of education indices 

94- 05 Cost-of-Education Differentials Across the States 

Course-taking 

95- 12 Rural Education Data User’s Guide 
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98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1 988 
1 999-05 Procedures Guide for Transcript Studies 
1 999-06 1 998 Revision of the Secondary School Taxonomy 

Crime 

97- 09 Status of Data on Crime and Violence in Schools: Final Report 

Curriculum 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

Customer service 

1 999- 1 0 What Users Say About Schools and Staffing Survey Publications 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1 999 ASA and 

1 999 AAPOR Meetings 



Data quality 

97-13 Improving Data Quality in NCES : Database-to-Report Process 
Data warehouse 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 



Design effects 

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 
Variances from NCES Data Sets 

Dropout rates, high school 

95- 07 National Education Longitudinal Study of 1 988: Conducting Trend Analyses HS&B and 

NELS:88 Sophomore Cohort Dropouts 

Early childhood education 

96- 20 1 991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96- 22 1 995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 

97- 24 Formulating a Design for the ECLS: A Review of Longitudinal Studies 

97- 36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood 

Programs: A Review and Recommendations for Future Research 

1 999- 0 1 A Birth Cohort Study: Conceptual and Design Considerations and Rationale 

Educational attainment 

98- 1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 

Test Report 

Educational research 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

Employment 

96-03 National Education Longitudinal Study of 1 988 (NELS:88) Research Framework and 
Issues 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 
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Faculty - higher education 

97- 26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

2000-01 1 999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

Finance - elementary and secondary schools 

94- 05 Cost-of-Education Differentials Across the States 

96- 1 9 Assessment and Analysis of School-Level Expenditures 

98- 01 Collection of Public School Expenditure Data: Development of a Questionnaire 

1 999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

1 999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model 
Approach 

Finance - postsecondary 

97- 27 Pilot Test of IPEDS Finance Survey 

Finance - private schools 

95- 17 Estimates of Expenditures for Private K-12 Schools 

96- 16 Strategies for Collecting Finance Data from Private Schools 

97- 07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 

Schools: An Exploratory Analysis 

97- 22 Collection of Private School Finance Data: Development of a Questionnaire 

1 999- 07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

Geography 

98- 04 Geographic Variations in Public Schools’ Costs 
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Inflation 

97-43 Measuring Inflation in Public School Costs 
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2000-01 1 999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

Instructional resources and practices 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
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1 999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test 
Results to Improve Item Construction 



International comparisons 
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Literacy of adults 

98-17 Developing the National Assessment of Adult Literacy: Recommendations from 
Stakeholders 

1999-09a 1992 National Adult Literacy Survey: An Overview 

1 999-09b 1992 National Adult Literacy Survey: Sample Design 

1 999-09c 1992 National Adult Literacy Survey: Weighting and Population Estimates 

1 999-09d 1992 National Adult Literacy Survey: Development of the Survey Instruments 

1999-09e 1992 National Adult Literacy Survey: Scaling and Proficiency Estimates 

1999-09f 1992 National Adult Literacy Survey: Interpreting the Adult Literacy Scales and Literacy 

Levels 

1999-09g 1992 National Adult Literacy Survey: Literacy Levels and the Response Probability 

Convention 

1999- 1 1 Data Sources on Lifelong Learning Available from the National Center for Education 

Statistics 

2000- 05 Secondary Statistical Modeling With the National Assessment of Adult Literacy: 

Implications for the Design of the Background Questionnaire 
2000-06 Using Telephone and Mail Surveys as a Supplement or Alternative to Door-to-Door 
Surveys in the Assessment of Adult Literacy 
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97-33 Adult Literacy: An International Perspective 
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Mathematics 

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in Jeffrey Owings 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test Dan Kasprzyk 

Results to Improve Item Construction 



Parental involvement in education 

96- 03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and Jeffrey Owings 

Issues 

97- 25 1996 National Household Education Survey (NHES:96) Questionnaires: Kathryn Chandler 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 
1 999-0 1 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West 



Participation rates 

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks Peter Stowe 
and Empirical Studies 



Postsecondary education 

1 999-1 1 Data Sources on Lifelong Learning Available from the National Center for Education Lisa Hudson 

Statistics 



Postsecondary education - persistence and attainment 
98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field Aurora D’Amico 
Test Report 

1 999-15 Projected Postsecondary Outcomes of 1 992 High School Graduates Aurora D’Amico 



Postsecondary education - staff 

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler 

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler 



Private schools 

96- 16 Strategies for Collecting Finance Data from Private Schools 
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Projections of education statistics 

1 999-15 Projected Postsecondary Outcomes of 1 992 High School Graduates 

Public school finance 

1 999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model 
Approach 



Public schools 

97- 43 Measuring Inflation in Public School Costs 

98- 01 Collection of Public School Expenditure Data: Development of a Questionnaire 

98-04 Geographic Variations in Public Schools* Costs 

1 999-02 Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results 

Public schools - secondary 

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

Reform, educational 

96-03 National Education Longitudinal Study of 1 988 (NELS:88) Research Framework and 
Issues 

Response rates 

98-02 Response Variance in the 1 993-94 Schools and Staffing Survey: A Reinterview Report 

School districts, public 

98-07 Decennial Census School District Project Planning Report 

1 999-03 Evaluation of the 1 996-97 Nonfiscal Common Core of Data Surveys Data Collection, 

Processing, and Editing Cycle 

School districts, public - demographics of 

96- 04 Census Mapping Project/School District Data Book 

Schools 

97- 42 Improving the Measurement of Staffing Resources at the School Level: The Development 

of Recommendations for NCES for the Schools and Staffing Survey (SASS) 

98- 08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 

1 999- 03 Evaluation of the 1 996-97 Nonfiscal Common Core of Data Surveys Data Collection, 

Processing, and Editing Cycle 

Schools - safety and discipline 

97-09 Status of Data on Crime and Violence in Schools: Final Report 

Software evaluation 

2000- 03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 

Variances from NCES Data Sets 

Staff 

97- 42 Improving the Measurement of Staffing Resources at the School Level: The Development 

of Recommendations for NCES for the Schools and Staffing Survey (SASS) 

98- 08 The Redesign of the Schools and Staffing Survey for 1 999-2000: A Position Paper 

Staff - higher education institutions 

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

State 

1 999-03 Evaluation of the 1 996-97 Nonfiscal Common Core of Data Surveys Data Collection, 

Processing, and Editing Cycle 
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Statistical methodology 

97-21 Statistics for Policymakers or Everything You Wanted to Know About Statistics But 
Thought You Could Never Understand 

Students with disabilities 

95- 13 Assessing Students with Disabilities and Limited English Proficiency 

Survey methodology 

96- 1 7 National Postsecondary Student Aid Study: 1 996 Field Test Methodology Report 

97- 1 5 Customer Service Survey: Common Core of Data Coordinators 

97- 35 Design, Data Collection, Interview Administration Time, and Data Editing in the 1 996 

National Household Education Survey 

98- 06 National Education Longitudinal Study of 1 988 (NELS:88) Base Year through Second 

Follow-Up: Final Methodology Report 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

98-16 A Feasibility Study of Longitudinal Design for Schools and Staffing Survey 
1 999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

1 999- 1 7 Secondary Use of the Schools and Staffing Survey Data 

2000- 01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

2000-02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1 999 ASA and 

1 999 AAPOR Meetings 

Teachers 

98-13 Response Variance in the 1994-95 Teacher Follow-up Survey 

1999-14 1994-95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook 

Teachers - instructional practices of 

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 

Teachers - opinions regarding safety 

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 

Teachers - performance evaluations 
1 999-04 Measuring Teacher Qualifications 

Teachers - qualifications of 

1 999- 04 Measuring Teacher Qualifications 

Teachers - salaries of 

94- 05 Cost-of-Education Differentials Across the States 

Variance estimation 

2000- 03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 

Variances from NCES Data Sets 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1 998 and 1 999 ASA and 
1 999 AAPOR Meetings 

Violence 

97-09 Status of Data on Crime and Violence in Schools: Final Report 

Vocational education 

95- 12 Rural Education Data User’s Guide 

1 999-05 Procedures Guide for Transcript Studies 

1 999-06 1 998 Revision of the Secondary School Taxonomy 
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